The present invention relates to methods and systems for imaging. More particularly, the present invention relates to methods and systems capable of capturing depth information in an image.
Since the advent of photography, significant advances have been made to improve the information that can be gathered from a picture. For example, certain photographic techniques enable three-dimensional (3D) information to be obtained by taking a number of different two-dimensional (2D) images at various viewpoints, which are then combined with certain algorithms to generate a 3D reconstruction of the scene. Effectively, in such cases, the lightfield of a scene is being sampled at different viewpoints and being combined to recover the 3D information. Many different methods of sampling lightfields are known (see for example S. M. Seitz and J. Kim, “The space of all stereo images,” International Journal of Computer Vision, 48:21-38, 2002; H.-Y. Shum and L.-W. He, “Rendering with concentric mosaics, Siggraph, 1999, which is hereby incorporated by reference herein in its entirety).
For example, one aspect of sampling the lightfield of a scene may include obtaining depth information, which can be carried out utilizing stereographic imaging techniques. In conventional techniques for stereographic imaging, two pictures are taken at the same time, one slightly to the side of the other—as though one camera was one eye and the other camera was another eye. These pictures are then displayed side by side and a 3D depth effect can be produced if the left eye looks only at the left image, and the right eye only looks at the right image.
Many stereographic imaging systems are designed, however, so that the geometrical properties of the optical systems are fixed before the lightfield of a scene is sampled. For example, the two-camera system described above utilizes a popular design wherein the epipolar lines are all parallel to each other and coincident with the horizontal scan lines of images.
Epipolar lines are lines that form on an image plane when epipolar planes intersect the image plane, wherein epipolar planes are defined as a plane containing the optical centers of the two cameras and any given point in the scene of interest.
Systems having such horizontal epipolar lines suffer from a serious problem. Horizontal edges are common in most real world scenes and their projections coincide with the epipolar lines. Thus, disparities along these edges cannot be computed, and stereo algorithms using these images are not able to compute depth at such points in space.
To alleviate such problems, panoramic stereo systems have incorporated an alternative geometry, wherein the epipolar lines are radial in the image (see for example J. Gluckman, S.K. Nayar, and K.J. Thoresz, “Real time omnidirectional and panoramic stereo,” DARPA Image Understanding Workshop, pages 299-303. November 1998; S.-S. Lin and R. Bajcsy, “High resolution catadioptric omni-directional stereo sensor for robot vision,” International Conference on Robotics and Automation, pages 1694-1699, 2003, which are hereby incorporated by reference herein in their entireties). Such geometry is not as likely to suffer from the afore-mentioned problem, as most objects do not possess purely radial edges. However, such panoramic stereo multi-camera systems are typically focused on outward looking views and have low spatial resolution.
In response to a demand for capturing larger amounts of information from images, catadioptric systems have been developed (see for example, J. Gluckman, S.K. Nayar, and K.J. Thoresz, “Real time omnidirectional and panoramic stereo,”DARPA Image Understanding Workshop, Pages 299-303, November 1998; C. Geyer and Kostas Daniilidis, “Structure and motion from uncalibrated catadioptric views,” IEEE Conference on Computer Vision and Pattern Recognition, pages 279-286, 2001; S.-S. Lin and R. Bajcsy, “High resolution catadioptric omni-directional stereo sensor for robot vision”, International Conference on Robotics and Automation, pages 1694-1699, 2003, which are hereby incorporated by reference herein in their entireties). Catadioptric systems are optical systems containing a combination of refracting and reflecting elements, such as lenses and mirrors, and have successfully been employed to increase the field of views of each camera used.
Nevertheless, multi-camera stereo systems, whether with or without mirrors, require geometric and photometric calibration, making their use time consuming, tedious, and prone to errors. These problems can be further exacerbated when dealing with dynamic scenes, as synchronization between the multiple cameras may be necessary.
To alleviate such problems, a number of techniques have been proposed to perform stereographic imaging using a catadioptric single camera system, wherein one or more mirrors are used to simulate virtual viewpoints (see for example S. Nene and S.K. Nayar, “Stereo with mirrors,” International Conference on Computer Vision, 1998; D. Southwell, A. Basu, M. Fiala, and J. Reyda, “Panoramic stereo,” International Conference on Pattern Recognition, pages 378-382, 1996; J. Gluckman and S.K. Nayar, “Planar catadioptric stereo; Geometry and calibration,” IEEE Conference on Computer Vision and Pattern Recognition, 1999, which are hereby incorporated by reference herein in their entireties). Advantages of such catadioptric single camera systems include requiring little or no calibration, being mobile, being easy to use, and having no need for synchronization when taking pictures of dynamic scenes. However, catadioptric single camera systems having radial epipolar geometry and the attendant advantages thereof have not yet been realized.